Performance Measures Used in Image Information Retrieval

Although during the running of the ImageCLEF tracks there was no explicit co–ordination on the types of evaluation measures employed, the same statistics were often used across ImageCLEF. Therefore, in this chapter, the range of measures used in the evaluation exercise is described. The original research defining a measure, together with their formulations and the relative pros and cons of the measures, are also detailed. Research that both compares the measures and attempts to determine the best is also outlined. Finally, the use of measures in the different tracks and years of ImageCLEF is tabulated. 5.1 Evaluation Measures Used in ImageCLEF At its most general, measurement has been described as the assignment of numerals to things so as to represent facts and conventions about them (Stevens, 1946). ImageCLEF is a collaborative evaluation exercise examining image searching or image analysis systems. In this context, the main purpose of measurement is as a means of comparison between different systems. More specifically, the evaluation acts as a simulation of a potential operational setting into which the systems may be deployed. The measures provide a way of determining which system submitted to a particular track of ImageCLEF would be best in the setting simulated by the test collection. Different measures reflect different priorities in the simulation of the operational setting. For example, in retrieval some evaluation measures stress the importance of locating as many relevant images as possible, while others focus on a small fixed size output. The choice of evaluation measure employed by different ImageCLEF track co–ordinators reflects varying priorities over the years of ImageCLEF. In this chapter, the descriptions of the measures are organized so as to Mark Sanderson University of Sheffield, Sheffield, United Kingdom e-mail: [email protected] H. Müller et al. (eds.), ImageCLEF, The Information Retrieval Series 32, DOI 10.1007/978-3-642-15181-1 5, © Springer-Verlag Berlin Heidelberg 2010 81